CCTV Pre-2024

Welcome to CCTV documentation page

• The documentation has been split into 2 parts - Preprocessing, and Training and prediction.
• Preprocessing section contains details on what data is used, where they are located, steps to run through the preprocessing pipeline and related information.
• Training and prediction section contains details on how to train the model and predict the output, along with the questions and replies from multiple emails aggregrated in one place.
• Note that the code for the same is mentioned in pages it is referred to.
• Most of the code to run this is located in cctv-apps, gqc-utility-notebooks, compute-canada and compute-msi.
  • Note that compute-canada might be obsolete once MSI is actively used and all code has been refactored to work for MSI.
• Active work is being done at the moment to refactor code for compute-msi and populating the respective repository with working code once we are able to run the whole pipeline successfully.

Product overview

There are 2 main products.

1. CCTV → Video frame classification for defect identification.

   1. Section preprocessing → Preparing a video dataset for training / prediction
   2. Section training-and-prediction → Training / Prediction
   3. Section CCTV Apps → Streamlit application containing different utilities

2. CCTVGIS / lightGBM → Binary model for predicting defects based on metadata of a pipe.

   1. Section lightGBM

Right now we are only updating the pipe-material in the GIS using the information from the metadata. We can update the pipe material from the information from video frames.

Information about the Metadata organization (Inspection Data):

The Inspection Data is given to us in different forms depending on the Utility and could be of the file type .csv, .dbf, .db, or .mdb. Regardless of the way the Inspection Information is stored, it has two tables: a Master table, and a Details table.

• Master: The Master table contains information about the inspection. For example, in the case of COV (where the inspection information is stored as a CSV) the Master table contains information about the Certificate Number, Surveyed By, Location, and information about the pipe inspected.
• Details: The Details table contains information about the types of Defects that were found during the inspection and it also contains the distance values indicating where the Defects occured.

The Master and Details section are linked to each other with a Inspection ID column (in the COV inspection data, the column is sometimes called cross_reference_id). A single Header is associated with only one Inspection ID, but the Details section can contain multiple rows with the same Inspection ID as there can be multiple defects within a single inspection.

Overall working of CCTV:

Supporting documentation

1. Data → details about data organization
2. ebmud → Details on processing done with ebmud dataset